Studies of model selection and regularization for generalization in neural networks with applications

This thesis investigates the generalization problem in artificial neural networks, attacking it from two major approaches: regularization and model selection. On the regularization side, under the framework of Kullback–Leibler divergence for feedforward neural networks, we develop a new formula for the regularization parameter in Gaussian density kernel estimation based on available training data sets. Experiments show that the estimated regularization parameter is valid for most cases. With the derived formula, all sample data sets can be used to estimate the smoothing parameter, which is less computationally expensive than using the leave-one-out cross-validation method. Furthermore, the new covariance matrix estimation formula is suitable for small sample data with high dimension setting in the regularized Gaussian classifier case. On the model selection side, both theory and extensive experiments are conducted for investigating the Bayesian Ying-Yang (BYY) model selection criterion to determine the cluster number in small sample-size cases. We derive new formula for estimating the smoothing parameters with proper approximations in the Smoothed ExpectationMaximum (SEM) technique. Experimental results show that with improved mixture model parameters, the BYY model selection criterion performance is enhanced. From the model selection viewpoint, generalization can also be improved by combining several nets to form ensemble networks. The relationship between Mixture of Experts (ME) and the ensemble networks is established in this thesis. In an approximation where ME reduces to ensemble neural nets, the ensemble nets can be globally optimized instead of being individual members. A new method is consequently proposed to average